Validation of the China mortality prediction model in trauma based on the ICD-10-CM codes

The China mortality prediction model in trauma, based on the International Classification of Diseases, Tenth Revision, Clinical Modification lexicon (CMPMIT-ICD-10), is a novel model for predicting outcomes in patients who experienced trauma. This model has not yet been validated using data acquired from patients at other trauma centers in China. This retrospective study used data retrieved from the Peking University People’s Hospital discharge database and included all patients admitted for trauma between 2012 and 2022 for model validation. Model performance was categorized into discrimination and calibration. In total, 23,299 patients were included in this study, with an overall mortality rate of 1.2%. CMPMIT-ICD-10 showed good discrimination and calibration, with an area under the curve of 0.84 (95% confidence interval: 0.82–0.87) and a Brier score of 0.02. The performance of the CMPMIT-ICD-10 during validation was satisfactory, and the application of the model will be scaled up in future studies.


Introduction
Trauma mortality prediction models are essential for trauma patient care, trauma-related clinical research, and the assessment of the quality of trauma care.Currently, the mortality risk from trauma is assessed using 2 types of models.[3] The second type includes models based on codes from the International Classification of Diseases, Tenth Revision, Clinical Modification lexicon (ICD-10-CM) such as the Trauma Mortality Prediction model based on ICD-10-CM Codes, Injury Mortality Prediction, and ICD-10-based ISS. [4,5]Although ISS has been widely used, coders require specialized training to use complex AIS codes, which is time-and resource-consuming. [5,6]Furthermore, most Chinese databases do not have separate AIS codes, whereas ICD-10-CM codes are widely used.The trauma mortality prediction model based on ICD-10-CM codes is relatively complex and does not account for factors such as previous comorbidities or acute physiological reactions.Therefore, to overcome the limitations of existing trauma mortality prediction models, Wang et al developed the China mortality prediction model for trauma based on ICD-10-CM codes (CMPMIT-ICD-10), which was the first prediction model established specifically for Chinese patients who experience trauma. [7]The predictor variables in the CMPMIT included sex, age, ICD-10-CM codes, comorbidities, post-traumatic physiological responses, and state of consciousness.CMPMIT has shown excellent performance in development datasets and internal validations; however, it has yet to be validated outside the developing environment.

Participants
Peking University People's Hospital is responsible for the treatment and care of trauma patients in Beijing and the surrounding areas.The hospital collects data on post-trauma patients from its discharge database, including patient demographic characteristics, ICD-10-CM codes, state of consciousness, mechanism of injury, treatment, expenses, and outcomes at discharge.
We included all patients admitted to the hospital for traumatic events between January 2012 and May 2022 as the study population.Traumatic events were those listed in Chapter 20 of the ICD-10-CM, excluding hanging, asphyxia, drowning, poisoning, burns, and electrocution.Patients with missing baseline characteristics or outcomes were excluded.This was a retrospective, non-interventional study based on anonymous registry data.

Statistical methods
The mortality risk associated with CMPMIT was calculated as described in our previous study.
where P (death) is the probability of death and e is the constant 2.718282.The value of b was calculated from the following equation: where Xi, i = 1...9 represents the 9 region-severity codes included in the model as the dichotomous variables, Ci, i = 1...5 represents the dichotomous variables for each comorbidity, "sex," "coma," and "traumatic shock" are dichotomous variables, with "male" as the reference group in sex, and "age" is a multicategorical variable.Mortality at discharge was considered as the outcome.R Studio 4.0.2 and SPSS v24.0.0.1 (IBM Corporation, Armonk) were used for statistical analysis.The variables included in the study were subjected to a univariate analysis.We used the chi-square test or Fisher exact test for categorical univariate analyses.We used the Mann-Whitney U test or Student t test for continuous data.
In this study, discrimination and calibration were used to assess the model performance.Discrimination assesses a model's ability to distinguish between deceased and surviving patients.The area under the receiver operating characteristic curve (AUC) can be used as a measure of discrimination, with an AUC closer to 1 indicating better discrimination.Generally, discrimination with an AUC > 0.90 is considered excellent, AUC > 0.80 is considered good, and AUC < 0.70 is considered poor.Discrimination is a key concern when assessing the model performance. [8]alibration reflects the accuracy of the model's absolute value of risk prediction; that is, the degree to which the model's predicted probability agrees with the observed probability of an event occurring.The Brier score was used as a measure of calibration and was calculated as follows: where Yt is the actual observed value of the variable indicating the outcome (0 or 1) and Pt is the prediction probability given by the prediction model.A Brier score closer to 0 indicated better model calibration and a Brier score > 0.25 suggests that the model had no predictive power. [9] Results During the study period, 23,299 trauma patients were included, with 286 deaths.The mean age of the patients was 58 years, 10,887 (46.7%) were men, and the overall mortality rate was 1.2%.The baseline patient characteristics are shown in Table 1.
CMPMIT showed good discrimination, with an AUC of 0.84 (95% confidence interval: 0.82-0.87),and the receiver operating characteristic curve is shown in Figure 1.
Calibration of the model was excellent with a Brier score of 0.02, indicating that the observed mortality rate was close to the predicted value.

Discussion
[12] Medical resources are relatively scarce in developing countries.Thus, early and accurate prediction of mortality can provide information for patient care stratification to ensure the appropriate allocation of medical resources. [13]Owing to differences in medical standards and resources, a distinct trauma scoring system should be developed for developing countries to ensure effective triage. [14]s the cost of medical care for trauma treatment increases, assessing the quality of care across trauma treatment facilities becomes increasingly important.This requires the adjustment of post-trauma mortality risk to be as accurate as possible. [15]omparing crude mortality rates across hospitals is meaningless without first adjusting for differences in patient composition, as the observed differences in mortality rates may simply be the result of differences in the trauma patient population across hospitals. [16]As different hospitals have different patient compositions, mortality risk adjustment levels the playing field and makes the performance of different hospitals comparable.Therefore, there is a need for validated mortality prediction models that can be used to adjust for differences in patient composition and disease severity across hospitals.
Developing trauma mortality prediction models is challenging because thousands of injury types and combinations exist.However, predicting many trauma combinations is challenging.Currently, the most widely applied ISS model addresses this problem to some extent by calculating the AIS score squared for the 3 most severely injured body regions. [17]However, inconveniently, ISS-and ISS-based trauma mortality prediction models can only be used when injuries are described using AIS codes.Injury descriptions require trained injury coders to review medical records and parse each injury into an AIS code.However, this process is time-consuming and expensive.In China, AIS injury coding is performed only in some trauma centers.Unlike AIS codes, ICD-10 codes are commonly assigned to all injuries sustained by inpatients with trauma.Thus, a trauma mortality prediction model based on ICD-10 codes allows the prediction of mortality without recoding.
For these reasons, in our previous study, we developed the CMPMIT-ICD-10, the first trauma mortality prediction model based on Chinese trauma databases and ICD-10 codes. [4]ompared to other trauma mortality prediction models based on ICD-10 codes, which only focus on injuries, CMPMIT-ICD-10 focuses more on the overall condition of patients with trauma.The influence of comorbidities and post-traumatic physiological responses on mortality was further considered in this new model.In addition, the model is simple and easy to use.Under the current trauma care system in China, it is suitable for early risk stratification of trauma patients and quality assessment of trauma centers.However, since its establishment, the CMPMIT-ICD-10 has not yet been validated with other databases in terms of portability and generalizability.In this study, we validated the CMPMIT-ICD-10 using the discharge database of Peking University People's Hospital.Overall, the validation results were satisfactory.The AUC generated during model validation was 0.84, which indicated that the performance was not as good as that during validation using the development dataset but also that the model had relatively good discrimination.The Brier score for the model validation was 0.02, which was similar to that for validation using the development dataset, indicating that the model had good calibration.These results indicate that this novel model is portable and generalizable.
The following reasons may explain why the model did not perform as well in this validation, as in the validation using the developed dataset.In this model, many ICD-10 codes share a common coefficient after reclassification, which may not be an accurate measurement of trauma severity.Moreover, posttrauma biochemical indicators such as hemoglobin and base deficit were not included in the model, which may have led to a less accurate prediction of mortality risk. [8,18]he biggest limitation of this study is that the validation database was a single-center database and had low mortality rates despite its large sample size.Results from the validation using this database may not accurately reflect the model's ability to predict mortality risk in a severely traumatized population.[21][22][23][24] However, it is still unclear whether machine learning models or traditional logistic regression models are more effective and accurate in predicting the mortality of trauma patients.Owing to the limitations of our database, we were unable to compare this model with existing machine learning models.
This model must be validated in numerous settings in future studies.In addition, the model must be continuously refined using new data and methods.

Conclusions
The CMPMIT-ICD-10 showed good discrimination and calibration in the validation; therefore, it will be improved and applied on a larger scale in the Chinese population.

Figure 1 .
Figure 1.Receiver operating characteristic (ROC) curve for validation of China mortality prediction model in trauma based on International Classification of Diseases, Tenth Revision, Clinical Modification codes (CMPMIT-ICD-10).

Table 1
Baseline characteristics of patients included in the study (2012-2022).
A2-G3 are the new region-severity code.www.md-journal.com